I want to develop a program that watches and traverses a rather big network for data. This data should then be available to search through with my program, maybe through a web interface or something.

My intuitive thought at the moment is to build an architecture kind of like the one Google has. One "front-end" (the Google Search page) which is essentially a regular web application and one "back-end" (which in Google's case traverses the web). Now for the hard part: If I decide to make such a system, how should communication be done between these parts?

One idea I had is to use some kind of database that both the back-end and front-end can access, but then comes the issues of concurrent writes and reads. Another issue with just using a database to communicate is that it makes it hard to "notify" the other part when something changes. Let's say that I want the "front-end" part to push changes to the UI when a change is noticed in the back-end. Then the back-end would have to have some way of notifying the front-end of this.